[ntuple] Support multiple column representations in the merger#22017
Open
silverweed wants to merge 14 commits into
Open
[ntuple] Support multiple column representations in the merger#22017silverweed wants to merge 14 commits into
silverweed wants to merge 14 commits into
Conversation
b2ae5fc to
1db6b5e
Compare
Test Results 23 files 23 suites 3d 17h 2m 21s ⏱️ For more details on these failures, see this check. Results for commit 7531fbf. ♻️ This comment has been updated with latest results. |
82e5299 to
d3efcfc
Compare
857d425 to
60b1bde
Compare
2 tasks
1d462ee to
a249301
Compare
a249301 to
ab08930
Compare
4697f42 to
db9c393
Compare
jblomer
reviewed
Jun 16, 2026
jblomer
left a comment
Contributor
There was a problem hiding this comment.
Thank you! That is a huge step forward.
With all the added complexity, I think we need to give the merger another fresh look and try reduce the logic a bit to smaller blocks. For a future PR.
We should remember to backport
- [ntuple] Fix RFieldFundamental::GenerateColumns using the wrong repr idx
- [ntuple] update merger test to make sure we test nRepetitions
- [ntuple] Clarify a bit RFieldBase::EntryToColumnElementIndex
- [ntuple] Introduce feature flag 0 (nested deferred columns)
- Patch up column ids in the header extension (part of
[ntuple] Support multiple column representations in the merger)
db9c393 to
9eaef1b
Compare
2 tasks
Contributor
Author
Split off those commits into #22655. |
df490f2 to
30a4004
Compare
30a4004 to
bcf53b2
Compare
2c49e98 to
df617cd
Compare
Instead of calling continue multiple times in the AddColumnFromField loop, just early return in case of projected fields.
We are currently serializing columns per-field, but in case of late
column extension this might result in inconsistent sorting of the columns
in the serialized footer.
e.g. assume you have fields "A" and "B", both late model extended, both
with a single column:
- col 0 -> field A, repr 0
- col 1 -> field B, repr 0
Now you add a new column representation to field "A"; this new column
has id 2:
- col 2 -> field A, repr 1
When serializing this RNTuple, all columns are written in the footer by
RNTupleSerialize::SerializeColumnsForFields(). Before this change, they
would end up on disk in order: [0, 2, 1].
This would corrupt the data by swapping the pages for columns 2 and 1.
After this change, they get written as [0, 1, 2] which is the correct
order.
Note that this exact case is tested in ntuple_merger in the unit test
MergeDeferredAdvanced.
df617cd to
89db1e7
Compare
- RPagePersistentSink::AddColumnRepresentation - RPagePersistentSink::AddAliasColumn Internal functionality to be used by the Merger. This entails 2 additional changes: - AddExtendedColumnRanges needs to be updated to handle the case where a column representation is added to a field during writing after some clusters have already been written; - ShiftAliasColumns needs to properly shift the ids of extended alias columns when called, otherwise a mismatch may happen when serializing the footer
Two field descriptors might legitimately have different numbers of elements in their LogicalColumnIds vectors due to having a different number of representations. The proper check to do is the one on the column cardinality, which makes sure that the same number of columns is actually used at a time.
This test checks two things: 1. that the fix applied by 14ade04 works properly (this is already checked by MergeDeferredAdvanced but that might not be the case anymore if we update the merger to change the column encodings depending on the output compression - in which case that test would not be adding a late extended column anymore); 2. that the merger properly handles fields with the same cardinality but different number of column representations
…ation Also set the anchor version in InitFromDescriptor().
clarify that it is only valid for the 0th representation index
89db1e7 to
7531fbf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This Pull request:
Significantly reworks the innards of the RNTupleMerger to support fast merging of fields with different but compatible column representations.
Basically it does two things:
A potentially negative consequence that we might want to revisit is that now the merger won't ever adapt the columns' splitness to the output compression (e.g. if merging changes the source compression from 0 to 505 it will still encode the columns as unsplit, and vice-versa). This will probably be readded in a future PR.
In order to achieve this, some new internal functionality had to be added, most notably
RPagePersistentSink::AddColumnRepresentation.Note that this PR is independent on #21740, which in fact might not be needed at all.
IMPORTANT
This PR introduces our first feature flag and thus the first bump to the specs' major version (1.1.0.0). This means we can now start producing RNTuples which cannot be read by older ROOT versions.
TODO
AddExtendedColumnRangesChecklist: